Skip to content

TRACING-5422: Add an example OpenTelemetry Collector CR with filelog receiver that parses OpenShift cluster logs #95995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

TRACING-5422: Add an example OpenTelemetry Collector CR with filelog receiver that parses OpenShift cluster logs #95995

wants to merge 1 commit into from

Conversation

andreasgerstmayr
Copy link
Contributor

@andreasgerstmayr andreasgerstmayr commented Jul 11, 2025 •
edited
Loading

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 11, 2025
@openshift-ci-robot
Copy link

openshift-ci-robot commented Jul 11, 2025 •
edited by openshift-ci bot
Loading

@andreasgerstmayr: This pull request references TRACING-5422 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.20.0" version, but no target version was set.

In response to this:

Version(s):
all supported OCP versions

Issue:
https://issues.redhat.com/browse/TRACING-5422

Link to docs preview:

QE review:

  • QE has approved this change.

Additional information:

cc @max-cx

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 11, 2025
@ocpdocs-previewbot
Copy link

ocpdocs-previewbot commented Jul 11, 2025 •
edited
Loading

🤖 Tue Jul 15 12:53:02 - Prow CI generated the docs preview:

https://95995--ocpdocs-pr.netlify.app/openshift-enterprise/latest/observability/otel/otel-collector/otel-collector-receivers.html

@openshift-ci-robot
Copy link

openshift-ci-robot commented Jul 11, 2025 •
edited by openshift-ci bot
Loading

@andreasgerstmayr: This pull request references TRACING-5422 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.20.0" version, but no target version was set.

In response to this:

Version(s):
all supported OCP versions

Issue:
https://issues.redhat.com/browse/TRACING-5422

Link to docs preview:
https://95995--ocpdocs-pr.netlify.app/openshift-enterprise/latest/observability/otel/otel-collector/otel-collector-receivers.html#filelog-receiver_otel-collector-receivers

QE review:

  • QE has approved this change.

Additional information:

cc @max-cx

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

IshwarKanse
IshwarKanse reviewed Jul 15, 2025
View reviewed changes
observability/otel/otel-collector/otel-collector-receivers.adoc
filelog:
include:
- /var/log/pods/*/*/*.log
exclude:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andreasgerstmayr I verified the docs PR, just one thing, should we add this to the exclude path.

        exclude:
        - /var/log/pods/*/otc-container/*.log
        - "/var/log/pods/*/*/*.gz"
        - "/var/log/pods/*/*/*.log.*"
        - "/var/log/pods/*/*/*.tmp"
        - "/var/log/pods/default_*/*/*.log"
        - "/var/log/pods/kube-*_*/*/*.log"
        - "/var/log/pods/kube_*/*/*.log"
        - "/var/log/pods/openshift-*_*/*/*.log"
        - "/var/log/pods/openshift_*/*/*.log"

The paths exclude infrastructure logs and only collects application logs. This is what the logging team is using in the vector config with a clusterlog forwarder created to forward application logs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if the customer wanted to scrape infrastructure or application logs.
The ticket was about assistance setting up the SCC (https://issues.redhat.com/browse/TRACING-5422).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ok, we should anyways exclude these.

        - /var/log/pods/*/otc-container/*.log
        - "/var/log/pods/*/*/*.gz"
        - "/var/log/pods/*/*/*.log.*"
        - "/var/log/pods/*/*/*.tmp"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 done

@IshwarKanse
Copy link
Member

Updated filelog test:
https://github.com/openshift/distributed-tracing-qe/tree/main/tests/e2e-otel/filelog

% chainsaw test tests/e2e-otel/filelog        
Version: v0.2.12
No configuration provided but found default file: .chainsaw.yaml
Loading config (.chainsaw.yaml)...
- Using test file: chainsaw-test
- TestDirs [tests/e2e-otel/filelog]
- SkipDelete false
- FailFast false
- ReportFormat ''
- ReportName 'chainsaw-report'
- Namespace ''
- FullName false
- IncludeTestRegex ''
- ExcludeTestRegex ''
- ApplyTimeout 10s
- AssertTimeout 6m0s
- CleanupTimeout 5m0s
- DeleteTimeout 5m0s
- ErrorTimeout 5m0s
- ExecTimeout 15s
- DeletionPropagationPolicy Background
- Parallel 4
- Template true
- NoCluster false
- PauseOnFailure false
Loading tests...
- filelog (tests/e2e-otel/filelog)
Loading values...
Running tests...
=== RUN   chainsaw
=== PAUSE chainsaw
=== CONT  chainsaw
=== RUN   chainsaw/filelog
=== PAUSE chainsaw/filelog
=== CONT  chainsaw/filelog
    | 13:38:35 | filelog | @chainsaw                                                                                       | CREATE    | OK    | v1/Namespace @ chainsaw-filelog
    | 13:38:35 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | TRY       | BEGIN |
    | 13:38:36 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | APPLY     | RUN   | security.openshift.io/v1/SecurityContextConstraints @ otel-clusterlogs-collector-scc
    | 13:38:36 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | PATCH     | OK    | security.openshift.io/v1/SecurityContextConstraints @ otel-clusterlogs-collector-scc
    | 13:38:36 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | APPLY     | DONE  | security.openshift.io/v1/SecurityContextConstraints @ otel-clusterlogs-collector-scc
    | 13:38:36 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | APPLY     | RUN   | opentelemetry.io/v1beta1/OpenTelemetryCollector @ chainsaw-filelog/clusterlogs
    | 13:38:37 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | CREATE    | OK    | opentelemetry.io/v1beta1/OpenTelemetryCollector @ chainsaw-filelog/clusterlogs
    | 13:38:37 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | APPLY     | DONE  | opentelemetry.io/v1beta1/OpenTelemetryCollector @ chainsaw-filelog/clusterlogs
    | 13:38:37 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | ASSERT    | RUN   | apps/v1/DaemonSet @ chainsaw-filelog/clusterlogs-collector
    | 13:38:38 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | ASSERT    | DONE  | apps/v1/DaemonSet @ chainsaw-filelog/clusterlogs-collector
    | 13:38:38 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | ASSERT    | RUN   | v1/Pod @ chainsaw-filelog/*
    | 13:38:39 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | ASSERT    | DONE  | v1/Pod @ chainsaw-filelog/*
    | 13:38:39 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | TRY       | END   |
    | 13:38:39 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | TRY       | BEGIN |
    | 13:38:39 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | APPLY     | RUN   | v1/ConfigMap @ chainsaw-filelog/app-log-plaintext-config
    | 13:38:39 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | CREATE    | OK    | v1/ConfigMap @ chainsaw-filelog/app-log-plaintext-config
    | 13:38:39 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | APPLY     | DONE  | v1/ConfigMap @ chainsaw-filelog/app-log-plaintext-config
    | 13:38:40 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | APPLY     | RUN   | v1/ReplicationController @ chainsaw-filelog/app-log-plaintext-rc
    | 13:38:40 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | CREATE    | OK    | v1/ReplicationController @ chainsaw-filelog/app-log-plaintext-rc
    | 13:38:40 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | APPLY     | DONE  | v1/ReplicationController @ chainsaw-filelog/app-log-plaintext-rc
    | 13:38:40 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | ASSERT    | RUN   | v1/ReplicationController @ chainsaw-filelog/app-log-plaintext-rc
    | 13:38:41 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | ASSERT    | DONE  | v1/ReplicationController @ chainsaw-filelog/app-log-plaintext-rc
    | 13:38:41 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | ASSERT    | RUN   | v1/Pod @ chainsaw-filelog/*
    | 13:38:42 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | ASSERT    | DONE  | v1/Pod @ chainsaw-filelog/*
    | 13:38:42 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | TRY       | END   |
    | 13:38:42 | filelog | Check the collected logs for the app-plaintext-logs                                             | TRY       | BEGIN |
    | 13:38:42 | filelog | Check the collected logs for the app-plaintext-logs                                             | CMD       | RUN   |
        === COMMAND
        /bin/sh -c ./check_logs.sh
    | 13:38:45 | filelog | Check the collected logs for the app-plaintext-logs                                             | SCRIPT    | LOG   |
        === STDOUT
        Starting continuous log checking for app-log-plaintext.log in collector instances...
        This will continue until logs are found or chainsaw timeout is reached...
        
        === Attempt 1 ===
        Checking logs in all collector instances...
        Found collector pods: clusterlogs-collector-29wr5 clusterlogs-collector-frz89
        Checking logs in pod: clusterlogs-collector-29wr5
        Checking logs in pod: clusterlogs-collector-frz89
        ✓ "log.file.path" found in clusterlogs-collector-frz89
        ✓ "SVTLogger" found in clusterlogs-collector-frz89
        ✓ "Body: Str(.*SVTLogger.*app-log-plaintext-" found in clusterlogs-collector-frz89
        ✓ "k8s.container.name: Str(app-log-plaintext)" found in clusterlogs-collector-frz89
        ✓ "k8s.namespace.name: Str(chainsaw-filelog)" found in clusterlogs-collector-frz89
        SUCCESS: All required log strings found in collector instance clusterlogs-collector-frz89!
        Found logs for app-log-plaintext.log in collector instances.
    | 13:38:45 | filelog | Check the collected logs for the app-plaintext-logs                                             | SCRIPT    | DONE  |
    | 13:38:45 | filelog | Check the collected logs for the app-plaintext-logs                                             | TRY       | END   |
    | 13:38:45 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | CLEANUP   | BEGIN |
    | 13:38:45 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | DELETE    | OK    | v1/ReplicationController @ chainsaw-filelog/app-log-plaintext-rc
    | 13:38:46 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | DELETE    | OK    | v1/ConfigMap @ chainsaw-filelog/app-log-plaintext-config
    | 13:38:46 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | CLEANUP   | END   |
    | 13:38:46 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | CLEANUP   | BEGIN |
    | 13:38:46 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | DELETE    | OK    | opentelemetry.io/v1beta1/OpenTelemetryCollector @ chainsaw-filelog/clusterlogs
    | 13:38:46 | filelog | Create OpenTelemetry Collector with FileLog receiver to collect logs from the OpenShift cluster | CLEANUP   | END   |
    | 13:38:46 | filelog | @chainsaw                                                                                       | CLEANUP   | BEGIN |
    | 13:38:47 | filelog | @chainsaw                                                                                       | DELETE    | OK    | v1/Namespace @ chainsaw-filelog
    | 13:39:29 | filelog | @chainsaw                                                                                       | CLEANUP   | END   |
--- PASS: chainsaw (0.00s)
    --- PASS: chainsaw/filelog (54.66s)
PASS
Tests Summary...
- Passed  tests 1
- Failed  tests 0
- Skipped tests 0
Done.

@andreasgerstmayr
TRACING-5422: Add an example OpenTelemetry Collector CR with filelog …
80d5a4b 
…receiver that parses OpenShift cluster logs

Signed-off-by: Andreas Gerstmayr <agerstmayr@redhat.com>
@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Jul 15, 2025

@andreasgerstmayr: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@IshwarKanse IshwarKanse IshwarKanse approved these changes

Assignees
No one assigned
Labels
jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@andreasgerstmayr @openshift-ci-robot @ocpdocs-previewbot @IshwarKanse